System and method for facilitating creation of an educational test based on prior performance with individual test questions

ABSTRACT

In certain implementations, creation of an educational test may be facilitated based on prior performance with individual test questions. A user input indicating a subject matter for a test, one or more question categories related to the subject matter, and a questionee group for which the test is intended may be received. A plurality of questions may be obtained based on the user input. For each question of the plurality of questions, a question performance metric value associated with the question (that is calculated based on prior performance of one or more questionees with the question) may be obtained. A test performance metric value associated with the test may be calculated based on the question performance metric values associated with at least some questions of the plurality of questions. The at least some questions and the test performance metric value may be provided for presentation on a user interface.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of pending U.S. patent applicationSer. No. 14/728,808, filed Jun. 2, 2015, entitled “SYSTEM AND METHOD FORFACILITATING CREATION OF AN EDUCATIONAL TEST BASED ON PRIOR PERFORMANCEWITH INDIVIDUAL TEST QUESTIONS”, which is hereby incorporated herein byreference in its entirety.

FIELD OF THE INVENTION

The invention relates to creation of an educational test andbenchmarking the test results, and, more particularly, to creation of aneducational test based on prior performance with individual testquestions, measurement of the performance of the individuals beingtested, the instructors, or the institutions, and benchmarking of thetest results.

BACKGROUND OF THE INVENTION

Among other advantages, standardized testing across various schools,districts, or states enable students of the various schools, districts,or states to be compared, and are objective in nature. The process ofcreating and administering typical standardized tests is, however,complex and expensive. Moreover, with typical standardized testing, testresults that are generally used to benchmark the performance of astudent on a test are limited to the test results of other students whotook the same test. While standardized testing provides a valuable meansfor the objective comparisons, the very low frequency of standardizedtesting makes them only snapshots in time. Standardized testing does notoffer a means for continuous monitoring and comparison of educationalprocess. These and other drawbacks exist.

SUMMARY OF THE INVENTION

The invention addressing these and other drawbacks relates to methods,apparatuses, and/or systems for facilitating creation of an educationaltest based on prior performance with individual test questions,measurement of the performance of the individuals being tested, theinstructors, or the institutions, and benchmarking of the test results.

In accordance with one aspect of the invention, an electronic testmanagement platform that allows instructors (or other user) toefficiently and effectively create tests and assess the performance oftheir students against other students (e.g., regardless of whether theother students have taken the same tests) may be provided.

In an implementation, question performance metric values associated withpotential questions for an educational test, a test performance metricvalue associated with the test (calculated based on the questionperformance metric values), or other performance information may beutilized to facilitate creation of the test. As an example, a databaseof questions may be queried based on a subject matter, a questioncategory, an intended questionee group, or other information to obtain aplurality of questions for a test. One or more of the plurality ofquestions (obtained from the database) may then be selected for the testbased on question performance metric values associated with thequestions, a test performance metric value associated with the test, orother performance information.

In an implementation, a test may be generated based on one or moreuser-specified test performance benchmark values for the test. As anexample, a user may specify an intended difficulty level for the test,an intended duration of the test, an intended question composition forthe test (e.g., with respect to percentage of each question category forthe test), a date and time for which the test will be administered, orother criteria. The test may be generated responsive to the userspecifying the test criteria.

In an implementation, question performance metric values may becalculated and/or updated for individual questions, and the calculatedand/or updated question performance metric values may be stored inassociation with their respective questions in a question database. Inan implementation, information regarding questionees who were given thequestions to answer, information regarding the performance of thequestionees with respect to the questions, or other information may bestored. The stored information may be utilized to calculate (or update)the question performance metric values for the questions. The questionperformance metric values may be stored in association with thequestions (e.g., as performance information in a testing managementdatabase). Stored information regarding a questionee may comprise aquestionee identifier (e.g., an anonymous identifier), one or morequestionee group which with the questionee is related (e.g., educationallevel, age group, school, school region, state, country, etc.), or otherinformation. Stored information regarding the performance of aquestionee with respect to a question may comprise whether thequestionee answered the question, how much time the questionee took toanswer the question, whether the questionee correctly answered thequestion, or other information.

In an implementation, a previously calculated performance metric valueassociated with a question may be updated based on performance of a newquestionee with the question. As an example, when a test (comprising thequestion) is provided to a questionee who has not previously beenpresented with the question, performance of the questionee with thequestion (and other questions on the test) may be monitored. Informationregarding the performance of the questionee may then be utilized toupdate performance metric values associated with the question.

In an implementation, performance of one or more questionees may beanalyzed and/or compared to performance of one or more otherquestionees. As an example, performance of each questionee on eachquestion of a test, each question category of the test, or the overalltest may be compared with performance of his/her peers within the samequestionee group that took the test (e.g., 6th grade students who tookthe test, 6th grade in the same state who took the test, 6th gradestudents within the same school who took the test, etc.) on eachquestion of a test, each question category of the test, or the overalltest, respectively.

In an implementation, performance of one or more instructors may beanalyzed and/or compared to performance of one or more otherinstructors. As an example, performance of instructors may be determinedbased on performance of the respective students of the instructors.Performance metric values associated with an instructor may comprise (1)average amounts of time that the instructor's students spent onparticular questions of a test, (2) average amounts of time that theinstructor's students spent on particular question categories of thetest, (3) an average amount of time that the instructor's students spenton the overall test, (4) average percentages of questions that theinstructor's students correctly answered in the particular questioncategories of the test, (5) an average percentage of questions that theinstructor's students correctly answered in the overall test, or (6)other performance metric values. Performance metric values associatedwith one or more other instructors (with which the performance metricvalues of the instructor may be compared) may comprise (1) averageamounts of time that the other instructors' students spent on theparticular questions, (2) average amounts of time that the otherinstructors' students spent on the particular question categories ofsimilar tests, (3) an average amount of time that the other instructors'students spent on the similar tests, (4) average percentages ofquestions that the other instructors' students correctly answered in theparticular question categories of the similar tests, (5) an averagepercentage of questions that the other instructors' students correctlyanswered in the similar tests, or (6) other performance metric values.

Various other aspects, features, and advantages of the invention will beapparent through the detailed description of the invention and thedrawings attached hereto. It is also to be understood that both theforegoing general description and the following detailed description areexemplary and not restrictive of the scope of the invention. As used inthe specification and in the claims, the singular forms of “a”, “an”,and “the” include plural referents unless the context clearly dictatesotherwise. In addition, as used in the specification and the claims, theterm “or” means “and/or” unless the context clearly dictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary illustration of a system for facilitatingcreation of an educational test based on prior performance withindividual test questions, in accordance with an aspect of theinvention.

FIGS. 2 and 3 are exemplary illustrations of user interfaces thatfacilitate creation of an education test based on prior performance withindividual test questions, in accordance with aspects of the invention.

FIG. 4 is an exemplary illustration of a user interfaces that facilitateanalysis and comparison of performance of questionees, in accordancewith an aspect of the invention.

FIG. 5 is an exemplary illustration of a flowchart of a method offacilitating creation of an educational test based on prior performancewith individual test questions, in accordance with an aspect of theinvention.

FIG. 6 is an exemplary illustration of a flowchart of a method ofproviding presentation of a test performance metric value associatedwith a test, in accordance with an aspect of the invention.

FIG. 7 is an exemplary illustration of a flowchart of a method ofproviding a comparison of performance of a questionee with performanceof one or more other questionees, in accordance with an aspect of theinvention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the implementations of the invention. It will beappreciated, however, by those having skill in the art that theimplementations of the invention may be practiced without these specificdetails or with an equivalent arrangement. In other instances,well-known structures and devices are shown in block diagram form inorder to avoid unnecessarily obscuring the implementations of theinvention.

Exemplary System Description

FIG. 1 is an exemplary illustration of a system 100 for facilitatingcreation of an educational test based on prior performance withindividual test questions, in accordance with an aspect of theinvention. As shown in FIG. 1, system 100 may comprise server 102 (ormultiple servers 102). Server 102 may comprise question managementsubsystem 112, test creation subsystem 114, test administrationsubsystem 116, performance assessment subsystem 118, or othercomponents.

System 100 may further comprise user device 104 (or multiple userdevices 104). User device 104 may comprise any type of mobile terminal,fixed terminal, or other device. By way of example, user device 104 maycomprise a desktop computer, a notebook computer, a netbook computer, atablet computer, a smartphone, a navigation device, an electronic bookdevice, a gaming device, or other user device. Users may, for instance,utilize one or more user devices 104 to interact with server 102 orother components of system 100. It should be noted that, while one ormore operations are described herein as being performed by components ofserver 102, those operations may, in some implementations, be performedby components of user device 104.

In some implementations, the various computers and subsystemsillustrated in FIG. 1 may comprise one or more computing devices thatare programmed to perform the functions described herein. The computingdevices may include one or more electronic storages (e.g., testingmanagement database 132, or other electric storages), one or morephysical processors programmed with one or more computer programinstructions, and/or other components. The computing devices may includecommunication lines, or ports to enable the exchange of information witha network or other computing platforms. The computing devices mayinclude a plurality of hardware, software, and/or firmware componentsoperating together to provide the functionality attributed herein to theservers. For example, the computing devices may be implemented by acloud of computing platforms operating together as the computingdevices.

The electronic storages may comprise non-transitory storage media thatelectronically stores information. The electronic storage media of theelectronic storages may include one or both of system storage that isprovided integrally (e.g., substantially non-removable) with the serversor removable storage that is removably connectable to the servers via,for example, a port (e.g., a USB port, a firewire port, etc.) or a drive(e.g., a disk drive, etc.). The electronic storages may include one ormore of optically readable storage media (e.g., optical disks, etc.),magnetically readable storage media (e.g., magnetic tape, magnetic harddrive, floppy drive, etc.), electrical charge-based storage media (e.g.,EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.),and/or other electronically readable storage media. The electronicstorages may include one or more virtual storage resources (e.g., cloudstorage, a virtual private network, and/or other virtual storageresources). The electronic storage may store software algorithms,information determined by the processors, information received from theservers, information received from client computing platforms, or otherinformation that enables the servers to function as described herein.

The processors may be programmed to provide information processingcapabilities in the servers. As such, the processors may include one ormore of a digital processor, an analog processor, a digital circuitdesigned to process information, an analog circuit designed to processinformation, a state machine, and/or other mechanisms for electronicallyprocessing information. In some implementations, the processors mayinclude a plurality of processing units. These processing units may bephysically located within the same device, or the processors mayrepresent processing functionality of a plurality of devices operatingin coordination. The processors may be programmed to execute computerprogram instructions to perform functions described herein of subsystems112-118 or other subsystems. The processors may be programmed to executecomputer program instructions by software; hardware; firmware; somecombination of software, hardware, or firmware; and/or other mechanismsfor configuring processing capabilities on the processors.

It should be appreciated that the description of the functionalityprovided by the different subsystems 112-118 described herein is forillustrative purposes, and is not intended to be limiting, as any ofsubsystems 112-118 may provide more or less functionality than isdescribed. For example, one or more of subsystems 112-118 may beeliminated, and some or all of its functionality may be provided byother ones of subsystems 112-118. As another example, additionalsubsystems may be programmed to perform some or all of the functionalityattributed herein to one of subsystems 112-118.

Attention will now be turned to a more detailed description of variousimplementations comprising one or more features related to creation ofan educational test based on prior performance with individual testquestions, measurement of the performance of the individuals beingtested, and benchmarking of the test results. It should be noted thatfeatures described herein may be implemented separately or incombination with one another.

Test Creation

In an implementation, question performance metric values associated withpotential questions for an educational test, a test performance metricvalue associated with the test (calculated based on the questionperformance metric values), or other performance information may beutilized to facilitate creation of the test. As an example, a questionperformance metric value associated with a question may comprise anestimated difficulty level assigned to the question (e.g., based on userratings by instructors, questionees, or other users, an average amountof time that questionees took to answer the question, a percentage ofquestionees who correctly answered the question, etc.), an averageamount of time that questionees took to correctly answer the question,an average amount of time that questionees took to incorrectly answerthe question, a percentage of questionees who correctly answered thequestion, a percentage of questionees who incorrectly answered thequestion, or other performance metric value. A test performance metricvalue associated with a test may comprise an estimated test difficultylevel assigned to the test (e.g., based on ratings by instructors,questionees, or other users, an average amount of time to complete thetest, a percentage of questionees who answered all questions on the testor a threshold number of questions on the test, raw test scores of thequestionees for the test, etc.), an estimated time to complete the test(e.g., based on an average amount of time to complete the test, apercentage of questionees who completed all questions on the test or athreshold number of questions on the test, a summation of the averagetimes for the questions on the test, etc.), or other performance metricvalues.

In an implementation, test creation subsystem 114 may receive a userinput related to creation of a test. The user input may indicate asubject matter, one or more question categories related to the subjectmatter, a questionee group for which the test is intended, or otherinformation for creating the test (e.g., an intended difficulty levelfor the test, the weight of each question category for the test, anintended duration of the test, a date and time on which the test is tobe administered, etc.). Based on the user input, test creation subsystem114 may obtain a plurality of questions that correspond to the subjectmatter, the question categories, the intended questionee group, etc. Inone scenario, for example, an instructor (or other user) may utilize atest creation user interface to select Mathematics as a subject matter(e.g., from a list of available subject matters) to be tested and 6thgrade students as a questionee group for which the test is intended.Responsive to the selection of the subject matter and the questioneegroup, a plurality of question categories related to Grade 6 Mathematicsmay be presented on the user interface. The available questioncategories may, for example, comprise exponents and square roots,decimals, fractions, pre-algebra, algebra, geometry, coordinate graphs,consumer math, data and graphs, statistics, or other questioncategories. When the instructor selects one or more of the availablequestion categories, a question database (e.g., separate or a part oftesting management database 132) may be queried with the criteria“Mathematics,” “Grade 6,” and the selected question categories to obtainquestions that correspond to the selected question categories and Grade6 Mathematics for presentation on the user interface.

For each question of a plurality of questions obtained as potentialquestions for a test, test creation subsystem 114 may obtain performanceinformation associated with the question. The performance informationmay comprise (for each question) one or more question performance metricvalues associated with the question. The question performance metricvalues may comprise an estimated difficulty level assigned to thequestion (e.g., based on user ratings, an average amount of time thatquestionees took to answer the question, a percentage of questionees whocorrectly answered the question, a percentage of questionees whoincorrectly answered the question, etc.), an average amount of time thatquestionees took to correctly answer the question, an average amount oftime that questionees took to incorrectly answer the question, apercentage of questionees who correctly answered the question, apercentage of questionees who incorrectly answered the question, orother performance metric values.

In an implementation, the plurality of questions that are obtained froma question database (based on a selected questionee group and/or otherfactors) may comprise one or more questions that correspond to aplurality of questionee groups (e.g., education levels, age group,etc.). As an example, each of the plurality of questionee groups maycorrespond to a different educational level (e.g., Grades K-12 or othereducational levels). In one use case, an Algebra question suitable forGrades 6-9 may obtained as a potential question for a test based on aninstructor's selection of Mathematics as a subject matter for the test,Algebra as a question category for the test, and Grade 6 as a questioneegroup for which the test is intended. Although the particular Algebraquestion may be suitable for the different grade levels, the sameAlgebra question may be associated with different question performancemetric values of a particular question metric for each of the differentgrade levels. In a further use case, for example, when presented as apotential question for the test, the Algebra question may be presentedwith a difficulty level of 3.2 as a result of the test being intendedfor 6th graders. However, if the Algebra question had been obtained fora test intended for 7th graders, the Algebra question may alternativelybe presented with a difficulty level of 2.8. Likewise, in some usecases, with respect to other performance metrics (e.g., average time tocorrect or incorrectly answer the question), the question may bepresented with a different performance metric value based on the gradelevel for which the test is intended.

In an implementation, question performance metric values associated witha question may be calculated based on prior performance of one or morequestionees with the question. In an implementation, the questionperformance metric values may be calculated with respect to a questioneegroup. As an example, a first difficulty level associated with thequestion and the 6th grade level may be calculated based on priorperformance of one or more 6th graders with the question (e.g., based onan average amount of time that 6th grade students took to answer thequestion, a percentage of 6th grade students who correctly answered thequestion, etc.). A second difficulty level associated with the questionand the 7th grade level may be calculated based on prior performance ofone or more 7th graders with the question. When the question ispresented as a potential question for a test intended for 6th graders,the question may be presented with the first difficulty level(associated with the question and the 6th grade level). On the otherhand, when the question is presented as a potential question for a testintended for 7th graders, the question may be presented with the seconddifficulty level (associated with the question and the 7th grade level).

As another example, a first average amount of time for 6th graders tocorrectly answer the question may be calculated based on the amounts oftime that 6th grade students (who correctly answered the question) tookto answer the question (e.g., when those 6th grade students werepreviously tested on the question). A second average amount of time for7th graders to correctly answer the question may be calculated based onthe amounts of time that 7th grade students (who correctly answered thequestion) took to answer the question (e.g., when those 7th graders werepreviously tested on the question). When the question is presented as apotential question for a test intended for 6th graders, the question maybe presented with the first average amount of time to indicate anestimated amount of time that the intended 6th grade student group willtake to correctly answer the question. On the other hand, when thequestion is presented as a potential question for a test intended for7th graders, the question may be presented with the second averageamount of time to indicate an estimated amount of time that the intended7th grade student group will take to correctly answer the question.

In an implementation, performance assessment subsystem 118 may calculatea test performance metric value associated with a test based on questionperformance metric values associated with questions for the test. Testcreation subsystem 114 may provide the questions and the testperformance metric value for presentation on a user interface. As anexample, with respect to FIG. 2, a plurality of questions (e.g.,questions QX1-QX7, questions QY-QY3, etc.) may be obtained from aquestion database (e.g., testing management database 132) as potentialquestions for a test based on a user selection of (1) a subject matterfor the test (e.g., Subject Matter A), (2) question categories (relatedto the subject matter) for the test (e.g., Question Category X, QuestionCategory Y, etc.), (3) a questionee group for which the test is intended(e.g., 6th grade students in a particular school region, students withina particular age group in a certain state), (4) an intended difficultylevel for the test (e.g., difficulty levels 1-5), (5) the weight of eachquestion category for the test (e.g., 40% of questions to be related toQuestion Category X, 30% of questions to be related to Question CategoryY, etc.), (5) an intended duration of the test, or (6) other criteria.As shown in FIG. 2, the plurality of questions may be presented on auser interface 200 along with their respective associated questionperformance metric values (e.g., difficulty level, average time tocorrectly answer, average time to incorrectly answer, etc.). As the userselects one or more of the questions to be added to the test (or removesquestions from the test), one or more test performance metric valuesassociated with the test may be calculated and presented on the userinterface 200.

In one scenario, with respect to FIG. 2, responsive to the addition ofquestions QX1, QX4, QX6, QX7, and QY2 to the test, a test difficultylevel is calculated for the test (e.g., a difficulty level of 3.2) basedon the individual question difficulty levels of questions QX1, QX4, QX6,QX7, and QY2 (e.g., based on a weighted mean of the question difficultylevels or other average of the question difficulty levels). In anotherscenario, responsive to the addition of the questions, an estimated timeto complete the test may be calculated based on the individual averagetimes for answering the respective added questions (e.g., the averagetimes to correctly answer the questions, the average times toincorrectly answer the questions, etc.), the individual percentages ofquestionees who correctly (or incorrectly) answered the respective addedquestions, etc. In other scenarios, other test performance metric valuesmay be calculated for the test (e.g., based on individual questionperformance metric values associated with the respective questions addedto the test).

In an implementation, upon obtaining a plurality of questions thatcorrespond to one or more user-specified criteria (e.g., a subjectmatter, related question categories, an intended questionee group, etc.)for a test, test creation subsystem 114 may provide the plurality ofquestion for presentation on a user interface. In response, testcreation subsystem 114 may receive one or more user inputs indicatingselection of one or more questions (of the plurality of questions) forinclusion in the test. Responsive to the selection, performanceassessment subsystem 118 may calculate one or more test performancemetric values associated with the test based on question performancemetric values associated with the selected questions. As an example,with respect to FIG. 2, a test difficulty level of 3.2 for a test and acurrent estimated time of 10 minutes and 8 seconds to complete the test(when the test includes the selected 5 questions) may be calculated (andpresented on user interface 200) responsive to the selection ofquestions QX1, QX4, QX6, QX7, and QY2 (for inclusion in the test). Inone use case, the question difficulty level for each of the selectedquestions may be utilized to calculate the test difficulty level. Inanother use case, the average amount of time previously taken byquestionees to correctly answer individual ones of the selectedquestions and/or the average amount of time previously taken byquestionees to incorrectly answer individual ones of the selectedquestions may be utilized to calculate the estimated time to completethe test.

In an implementation, the test performance metric values may becalculated for the test in real-time as questions are selected forinclusion in the test. As an example, with respect to FIG. 2, the testdifficulty level of 3.2 and the estimated completion time of 10 minutesand 8 seconds may be calculated for the test responsive to a userselecting questions QX1, QX4, QX6, QX7, and QY2 to be added to the test.When the user selects an additional question to be added to the test,performance assessment subsystem 118 may calculate an updated testdifficulty level for the test based on the question difficulty levelsassociated the previously added questions and the question difficultylevel associated with the additional question. Performance assessmentsubsystem 118 may also calculate an updated estimated completion timefor the test based on the average completion times associated with thepreviously added questions and the average completion times associatedwith the additional question. Test creation subsystem 114 may thenprovide the updated test difficulty level, the updated estimatedcompletion time, and the updated selected set of questions forpresentation on user interface 200.

In an implementation, test creation subsystem 114 may assess one or moretest criteria to be satisfied, and may notify a user regarding whetherthere are test criteria that have not been satisfied along with one ormore actions that can be taken to satisfy unmet test criteria. As anexample, the test criteria to be satisfied for a test may comprisedefault test criteria for the test, user-specified test criteria for thetest, or other test criteria. Additionally, or alternatively, testcriteria for the test may be inferred based on user preferences, ainstructor's syllabus and/or class materials (e.g., homework previouslygiven during a semester or other school time period, quizzes or othertests previously given during the semester or other school time period,etc.), or other test criteria.

In one scenario, with respect to FIG. 2, during creation of the test, auser may specify weights of the question categories (e.g., specifyingthe percentages of the respective question categories) that are to beincluded in the test (e.g., 50% of questions related to QuestionCategory X, 30% of question related to Question Category Y, 20% ofquestions related to Question Category Z, etc.). As shown in FIG. 2, asthe user selects questions to be added to the test, an indication of thetypes of questions that should be further added to the test may bepresented on user interface 200. The types of questions (that should befurther added to the test) may comprise questions that are needed tosatisfy a question composition corresponding to the user-specifiedweighted question categories. For example, after the user selectsquestions QX1, QX4, QX6, QX7, and QY2 to be included for the test, thequestion composition may comprise 80% Question Category X questions and20% Question Category Y questions. As such, as shown in FIG. 2, anindication that questions related to Question Category Y and questionsrelated to Question Category Z should be added to the test to satisfythe question composition corresponding to the user-specified weightedquestion categories. It should be noted that a question may be relatedto one or more question categories. By adding a question to a test, eachof the question categories related to the added question may, forexample, be increased in percentage for the question composition of thetest.

In another scenario, a recommended question composition for a test(e.g., comprising respective percentages of questions related to variousquestion categories) intended for students in a particular class may becalculated based on the composition of the question categories inmaterials previously given to the students in the class. As an example,test creation subsystem 114 may assess the composition of questioncategories in a syllabus provided to the students in the class during asemester, the composition of question categories in homework previouslygiven to the students during the semester, the composition of questioncategories in quizzes or other tests previously given to the studentsduring the semester, or other composition of question categories inother class materials provided during the semester. Test creationsubsystem 114 may then calculate the recommended question compositionfor the test (e.g., a mid-semester exam, a final exam, etc.) based onthe compositions of question categories in the class materials, andutilize the recommended question composition to recommend in real-timethe types of questions that should be added to the test to satisfy therecommended question composition (e.g., as a user is adding questions tothe test, the recommendations regarding the types of questions to befurther added to the test may be updated in real-time).

User-Specified Test Benchmarks

In an implementation, test creation subsystem 114 may generate a testbased on one or more user-specified test performance benchmark valuesfor the test. As an example, a user may specify an intended difficultylevel for the test, an intended duration of the test, an intendedquestion composition for the test (e.g., with respect to percentage ofeach question category for the test), a date and time for which the testwill be administered, or other criteria. Responsive to the userspecifying the test criteria, test creation subsystem 114 may generatethe test and/or provide the test to test administration subsystem 116without further user input (e.g., such that, without further user input,the questions are automatically selected for the test and the test isprovided to students on the specified date and time).

As another example, with respect to FIG. 3, a user may specify anintended difficulty level for the test, and an intended duration of thetest, or other test performance benchmark values for the test. Based onthe user-specified test performance benchmark values, test creationsubsystem 114 may generate one or more question sets for the test. Eachof the question sets may, for example, comprise a complete set ofrecommended questions for the test. In one scenario, as shown in FIG. 3,the user may specify that he/she wants a test to have a difficulty levelabout 3.5 and a completion time of about 1 hour. Based on these criteria(along with one or more other criteria, such as a subject matter,question categories, the intended questionee group, etc.), test creationsubsystem 114 may generate Question Set 1, Question Set 2, Question Set3, Question Set 4, Question Set 5, etc., and present the generatedquestion sets on user interface 300 to enable the user to: (1) generatethe test so that the test comprises one of the generated question sets,(2) view one of the question sets, (3) modify one of the question setsso that the test comprises the modified question set (e.g., selectanother question from a question database to add to the question set,make up a new question to add to the question set, etc.), (4) or performother operations. As shown in FIG. 3, each of the question sets may begenerated such that the question set has test performance metric valuesthat correspond to the user-specified test performance benchmark values.For example, test creation subsystem 114 may select questions for eachof the question sets based on their respective question performancemetric values to generate question sets having test performance metricvalues that are close to the user-specified test performance benchmarkvalues.

In another scenario, the user may specify that he/she wants a test tohave certain question categories and the weights of the respectivequestion categories for the test. Based on the user-specified questioncategories and weights, test creation subsystem 114 may generate one ormore question sets for the test. With respect to FIG. 3, for example,based on the user-specified test criteria (e.g., test difficulty level,completion time, question categories and associated weights, etc.), testcreation subsystem 114 may generate question sets that correspond to theuser-specified test criteria. For example, test creation subsystem 114may select questions for each of the question sets based on theirrespective question performance metric values and their associatedquestion categories to generate question sets having: (1) testperformance metric values that are close to the user-specified testperformance benchmark values and (2) a question composition that closelyreflects the user-specified weights for the question categoriesspecified for the test.

In a further scenario, test creation subsystem 114 may rank the questionsets (that are generated based on the user-specified criteria). Theranking may be based on the test performance metric values associatedwith the question sets, the question compositions associated with thequestion sets, or other factors. As shown in FIG. 3, for example, thequestion sets may be ranked based on the closeness of test performancemetric values (e.g., test difficult levels, test completion times, etc.)of the question sets to the user-specified test performance benchmarkvalues (e.g., test difficulty benchmark level, test benchmark completiontime, etc.).

Question Performance Metric Value Updates

In an implementation, question performance metric values may becalculated and/or updated for individual questions, and the calculatedand/or updated question performance metric values may be stored inassociation with their respective questions in a question database(e.g., storing questions 134 and associated performance information 136in testing management database 132). In an implementation, questionmanagement subsystem 112 may store information regarding questionees whowere given the questions to answer, information regarding theperformance of the questionees with respect to the questions, or otherinformation. Performance assessment subsystem 118 may utilize the storedinformation to calculate (or update) the question performance metricvalues for the questions. Question management subsystem 112 may storethe question performance metric values in association with the questions(e.g., as performance information 136 in testing management database132). Stored information regarding a questionee may comprise aquestionee identifier (e.g., an anonymous identifier), one or morequestionee group which with the questionee is related (e.g., educationallevel, age group, school, school region, state, country, etc.), or otherinformation. Stored information regarding the performance of aquestionee with respect to a question may comprise whether thequestionee answered the question, how much time the questionee took toanswer the question, whether the questionee correctly answered thequestion, or other information.

As an example, a question difficulty level may be calculated for aquestion based on how long each questionee took to answer the question(e.g., an average amount of time), whether the question was answeredcorrectly by the questionee, how many of the questionees correctlyanswered the question (e.g., a percentage of questionees who correctlyanswered the question), how many of the questionees didn't provide ananswer the question, or other factors. An average amount of time forcorrectly answering the question may be calculated by determining amean, medium, or mode of the amounts of time that questionees previouslytook to correctly answer the question. An average amount of time forincorrectly answering the question may be calculated by determining amean, medium, or mode of the amounts of time that questionees took toincorrectly answer the question.

As a further example, a question difficulty level for a question, anaverage amount of time for correctly answering the question, an averageamount of time for incorrectly answering the question, or other questionperformance metric values may be calculated with respect to a particularquestionee group. In one use case, if the questionee group are 6th gradestudents, a question difficulty level associated with the question andthe questionee group may calculated based on how long each 6th gradestudent took to answer the question, whether the question was correctlyanswered by the 6th grade student, how many of the 6th grade studentscorrectly answered the question, how many of the 6th grade studentsdidn't provide an answer the question, or other factors. As such, forexample, when creating a test for 6th grade students, the questiondifficulty levels calculated with respect to 6th grade students may beutilized to facilitate the creation of the test. Likewise, in anotheruse case, an average amount of time for 6th grade students to correctlyanswer the question may be calculated by determining a mean, medium, ormode of the amounts of time that 6th grade students previously took tocorrectly answer the question. An average amount of time for 6th gradestudents to incorrectly answer the question may be calculated bydetermining a mean, medium, or mode of the amounts of time that 6thgrade students took to incorrectly answer the question. When creating atest for 6th grade students, the foregoing averages with respect to 6thgrade students may be utilized to facilitate the creation of the test.

In an implementation, performance assessment subsystem 118 may update apreviously calculated performance metric value associated with aquestion based on performance of a new questionee with the question. Asan example, when a test (comprising the question) is provided to aquestionee who has not previously been presented with the question,performance of the questionee with the question (and other questions onthe test) may be monitored. Information regarding the performance of thequestionee may then be utilized to update performance metric valuesassociated with the question. In one scenario, a question difficultylevel associated with the question may be updated by recalculating thequestion difficulty level using historical information regardingperformance of previous questionees with the question and theinformation regarding the performance of the questionee with thequestion. If the questionee answered the question correctly, an averageamount of time for correctly answering the question (e.g., previouslycalculated based on amounts of time that previous questionees took tocorrectly answer the question) may be updated by recalculating theaverage amount of time for correctly answering the question using theamounts of time that previous questionees took to correctly answer thequestion and the amount of time that the questionee took to correctlyanswer the question. If the questionee answered the questionincorrectly, an average amount of time for incorrectly answering thequestion may be updated by recalculating the average amount of time forincorrectly answering the question using the amounts of time thatprevious questionees took to incorrectly answer the question and theamount of time that the questionee took to incorrectly answer thequestion.

In an implementation, performance assessment subsystem 118 may update apreviously calculated question performance metric value associated witha question based on the previously calculated performance metric valueand performance of a new questionee with the question. As an example, atleast some information previously used to calculate the associatedperformance metric value need not be used to recalculate an updatedversion of the performance metric value for the question. As such,question performance metric values associated with questions may beupdated more efficiently.

As another example, a first weight may be assigned to one or morepreviously calculated question performance metric values associated withthe question, and a second weight may be assigned to performance of thequestionee with the question. In one use case, a first weighted questiondifficulty level may be calculated based on the first weight and apreviously calculated question difficulty level associated with thequestion. A second weighted question difficulty level may be calculatedbased on the second weight and an estimated level of difficulty that thequestionee had with the question (e.g., calculated based on whether thequestionee answered the question correctly, an amount of time that thequestionee took to answer the question, etc.). The updated questiondifficulty level may be calculated for the question based on the firstweighted question difficulty level and the second weighted questiondifficulty level. In some use cases, the first weight may be greaterthan the second weight (or vice versa). For example, the first weightmay be equal to the number of questionees whose performance thepreviously calculated question difficulty level is based on, and thesecond weight may be 1 to reflect the single questionee whoseperformance the estimated level of difficulty (of the questionee withthe question) is based on.

In another use case, where the questionee correctly answered thequestion, a first weighted average amount of time for correctlyanswering the question may be calculated based on the first weight and apreviously calculated average amount of time for correctly answering thequestion. A second weighted amount of time for correctly answering thequestion may be calculated based on the second weight and an amount oftime that the questionee took to correctly answer the question. Theupdated average amount of time for correctly answering the question maybe calculated based on the first weight average amount of time and thesecond weighted amount of time.

In a further use case, where the questionee incorrectly answered thequestion, a first weighted average amount of time for incorrectlyanswering the question may be calculated based on the first weight and apreviously calculated average amount of time for incorrectly answeringthe question. A second weighted amount of time for incorrectly answeringthe question may be calculated based on the second weight and an amountof time that the questionee took to incorrectly answer the question. Theupdated average amount of time for incorrectly answering the questionmay be calculated based on the first weight average amount of time andthe second weighted amount of time.

Questionee Performance Analysis

In an implementation, performance of one or more questionees may beanalyzed and/or compared to performance of one or more otherquestionees. As an example, performance assessment subsystem 118 maycompare performance of each questionee on each question of a test, eachquestion category of the test, or the overall test with performance ofhis/her peers within the same questionee group that took the test (e.g.,6th grade students who took the test, 6th grade in the same state whotook the test, 6th grade students within the same school who took thetest, etc.) on each question of a test, each question category of thetest, or the overall test, respectively.

In one scenario, with respect to FIG. 4, user interface 400 mayillustrate such a comparison where bars 402 a-402 f may representperformance metric values associated with Questionee X and bars 404a-404 f may represent performance metric values associated with an“average” student. The performance metric values associated withQuestionee X may, for example, comprise (1) amounts of time thatQuestionee X spent on particular questions (e.g., where at least one ofbars 402 a-402 f represents an amount of time spent on one particularquestion), (2) amounts of time that Questionee X spent on particularquestion categories of the test (e.g., where at least one of bars 402a-402 f represents an amount of time spent on one particular questioncategory), (3) an amount of time that Questionee X spent on the overalltest, (4) percentages of questions that Questionee X correctly answeredin the particular question categories of the test, (5) a percentage ofquestions that Questionee X correctly answered in the overall test, or(6) other performance metric values. The performance metric valuesassociated with the “average” student may comprise (1) average amountsof time that one or more other questionees (who took the test) withinthe same questionee group spent on the particular questions (e.g., whereat least one of bars 404 a-404 f represents an average amount of timespent on one particular question, (2) average amounts of time that theother questionees spent on the particular question categories of thetest (e.g., where at least one of bars 404 a-404 f represents an averageamount of time spent on one particular question category), (3) anaverage amount of time that the other questionees spent on the test, (4)average percentages of questions that the other questionees correctlyanswered in the particular question categories of the test, (5) anaverage percentage of questions that the other questionees correctlyanswered in the overall test, or (6) other performance metric values.Bars 402 a-402 f may indicate absolute performance metric values orrelative performance of Questionee X with respect to the performance ofthe “average” student. Likewise, bars 404 a-404 f may indicate absoluteperformance metric values or relative performance of the “average”student with respect to the performance of Questionee X. Othercomparison techniques may additionally or alternatively be utilized(e.g., percentile comparison).

As another example, performance assessment subsystem 118 may compareperformance of a questionee on a question of a test or a questioncategory of the test with performance of one or more other questioneeson the question or the question category even if the other questioneesdid not take the same test (e.g., the other questionees may have takenone or more different tests where each test comprises a different set ofquestions). With respect to FIG. 4, for example, user interface 400 mayillustrate such a comparison where bars 402 a-402 f may representperformance metric values associated with Questionee X and bars 404a-404 f may represent performance metric values associated with an“average” student. The performance metric values associated withQuestionee X may comprise (1) amounts of time that Questionee X spent onparticular questions of the test (e.g., where at least one of bars 402a-402 f represents an amount of time spent on one particular question),(2) amounts of time that Questionee X spent on particular questioncategories of the test (e.g., where at least one of bars 402 a-402 frepresents an amount of time spent on one particular question category),(3) an amount of time that Questionee X spent on the overall test, (4)percentages of questions that Questionee X correctly answered in theparticular question categories of the test, (5) a percentage ofquestions that Questionee X correctly answered in the overall test, or(6) other performance metric values. The performance metric valuesassociated with the “average” student may comprise (1) average amountsof time that the other questionees spent on the particular questions(e.g., where at least one of bars 404 a-404 f represents an averageamount of time spent on one particular question), (2) average amounts oftime that the other questionees spent on the particular questioncategories of a similar test (e.g., similar in difficulty, duration,question category composition, etc.), (3) an average amount of time thatthe other questionees spent on the overall similar test, (4) averagepercentages of questions that the other questionees correctly answeredin the particular question categories of the similar test, (5) anaverage percentage of questions that the other questionees correctlyanswered in the overall similar test, or (6) other performance metricvalues. In a further example, the other questionees (whose performanceis compared with Questionee X's performance) may be selected (forcomparison with Questionee X) based on a determination that thequestionees are in the same or similar questionee group as Questionee X.Questionee X may, for example, be 6th grade student in a first school ina first state, and the other questionees may comprise 6th grade studentsin other schools across numerous states.

As yet another example, performance assessment subsystem 118 may compareperformance of a group of questionees on a question with performance ofone or more other questionees on the question (e.g., even if thequestionees in the group of questions did not take the same test as oneanother, even if the other questionees did not take the same test asindividual questions in the group of questionees, etc.). In onescenario, for example, the performance metric values of the group ofquestionees (to be compared with the performance metric values of theother questionees) may comprise (1) average amounts of time that thequestionees in the group of questionees spent on particular questions,(2) average amounts of time that the questionees in the group ofquestionees spent on particular question categories of similar tests,(3) an average amount of time that the questionees in the group ofquestionees spent on the similar tests, (4) average percentages ofquestions that the questionees in the group of questionees correctlyanswered in the particular question categories of the similar tests, (5)an average percentage of questions that the questionees in the group ofquestionees correctly answered in the similar tests, or (6) otherperformance metric values. The performance metric values associated withthe other questionees may comprise (1) average amounts of time that theother questionees spent on the particular questions, (2) average amountsof time that the other questionees spent on the particular questioncategories of the similar tests, (3) an average amount of time that theother questionees spent on the similar tests, (4) average percentages ofquestions that the other questionees correctly answered in theparticular question categories of the similar tests, (5) an averagepercentage of questions that the other questionees correctly answered inthe similar tests, or (6) other performance metric values.

In an implementation, performance of one or more instructors may beanalyzed and/or compared to performance of one or more otherinstructors. As an example, performance of instructors may be determinedbased on performance of the respective students of the instructors.Performance metric values associated with an instructor may comprise (1)average amounts of time that the instructor's students spent onparticular questions of a test, (2) average amounts of time that theinstructor's students spent on particular question categories of thetest, (3) an average amount of time that the instructor's students spenton the overall test, (4) average percentages of questions that theinstructor's students correctly answered in the particular questioncategories of the test, (5) an average percentage of questions that theinstructor's students correctly answered in the overall test, or (6)other performance metric values. Performance metric values associatedwith one or more other instructors (with which the performance metricvalues of the instructor may be compared) may comprise (1) averageamounts of time that the other instructors' students spent on theparticular questions, (2) average amounts of time that the otherinstructors' students spent on the particular question categories ofsimilar tests, (3) an average amount of time that the other instructors'students spent on the similar tests, (4) average percentages ofquestions that the other instructors' students correctly answered in theparticular question categories of the similar tests, (5) an averagepercentage of questions that the other instructors' students correctlyanswered in the similar tests, or (6) other performance metric values.

In one scenario, a teacher can log into a testing management system (ora performance assessment subsystem of the testing management system) andrun analysis on his/her students' performance. Based on the analysis,the testing management system may provide a graphical representation ofeach student's performance (e.g., related to particular questions,particular question categories, etc.), a graphical comparison of thestudent's performance against other students (e.g., FIG. 4), or othergraphical representations related to the performance of the teacher'sstudents. In this way, the teacher may quickly and easily identify thestrengths and weaknesses of each of his students (e.g., by questioncategory or other criteria). Additionally, or alternatively, the teachermay quickly and easily identify strengths and weaknesses in his/her ownteaching techniques and identify areas of improvement that he/she needsto make.

In another scenario, the teacher may run comparative analysis betweenhis own performance as a teacher and other teachers (e.g., of the samegrade and subject that use the same text book within the same school orany other school). For example, the testing management system mayprovide the teacher with a graphical comparison of an averageperformance of the teacher's students (e.g., on particular questions, onparticular question categories of a test, or on the overall test) withan average performance of one or more other teachers' students (e.g., onthe particular questions, the particular question categories of asimilar test, or on the overall similar test). In a further scenario,the average performance comparison may compare (1) an averageperformance of a top percentile of the teacher's students (e.g., duringa semester or other time period) with an average performance of the toppercentile of one or more teachers' students, (2) average performance ofa bottom percentile of the teacher's students with an averageperformance of a bottom percentile of the other teacher's students, (3)or other averages. Additionally, or alternatively, the testingmanagement system may provide the teacher with a performance ratingassociated with the teacher (e.g., for a particular test, an overallsemester or other time period, etc.), performance ratings associatedwith the other teachers, or other performance ratings. The performanceratings of the teacher and the other teachers may, for example, bedetermined based on the performance of their respective students.

In yet another scenario, school administrations can log into the testingmanagement system and monitor the progress and performance of eachteacher, class, section, and/or student at any time without having toenter a single classroom. The testing management system may also enablethem to perform comparative analysis between sections, classes, and/orteachers within the same school or other schools. The testing managementsystem may further enable them to easily identify whether a teacher isproviding relatively easy questions to his/her students and whether thestudents are achieving average or below average results against thoseeasy questions, or whether the teacher is truly giving the studentschallenging questions and whether the students are demonstratingattainment of the material by performing at or above average against thechallenging questions. With such a testing management system, schooladministrations may quickly and easily identify their own strengths andweaknesses compared to other schools (e.g., of the same curriculum bygrade and subject). As such, they can identify precisely and objectivelywhat talent they need to attract to improve the quality of education intheir institution, market themselves as a magnet school in a particularfield (e.g., science, languages, arts, or other field depending on theirrelative strengths and weaknesses), etc.

It should be noted that, although some implementations described hereinare with respect analysis and/or comparison of performance ofinstructors based on the performance of their respective students, it isunderstood that (to the extent possible) analysis and/or comparison ofother entities (e.g., schools or other institutions, city, state, orother governing region, etc.) may also be based on the performance oftheir respective students (e.g., students studying at the institutions,students studying in the governing region, etc.). As an example,performance metric values associated with an entity may comprise (1)average amounts of time that the entity's students spent on particularquestions, (2) average amounts of time that the entity's students spenton particular question categories of similar tests, (3) average amountsof time that the entity's students spent overall on the similar tests,(4) average percentages of questions that the entity's studentscorrectly answered in the particular question categories of the similartests, (5) average percentage of questions that the entity's studentscorrectly answered in the similar tests, or (6) other performance metricvalues.

Exemplary Flowcharts

FIGS. 5-7 comprise exemplary illustrations of flowcharts of processingoperations of methods that enable the various features and functionalityof the system as described in detail above. The processing operations ofeach method presented below are intended to be illustrative andnon-limiting. In some implementations, for example, the methods may beaccomplished with one or more additional operations not described,and/or without one or more of the operations discussed. Additionally,the order in which the processing operations of the methods areillustrated (and described below) is not intended to be limiting.

In some implementations, the methods may be implemented in one or moreprocessing devices (e.g., a digital processor, an analog processor, adigital circuit designed to process information, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information). The one or moreprocessing devices may include one or more devices executing some or allof the operations of the methods in response to instructions storedelectronically on an electronic storage medium. The one or moreprocessing devices may include one or more devices configured throughhardware, firmware, and/or software to be specifically designed forexecution of one or more of the operations of the methods.

FIG. 5 is an exemplary illustration of a flowchart 500 of a method offacilitating creation of an educational test based on prior performancewith individual test questions, in accordance with an aspect of theinvention.

In an operation 502, a first user input (indicating a subject matter fora test, one or more question categories related to the subject matter,and a questionee group for which the test is intended) may be received.Operation 502 may be performed by a test creation subsystem that is thesame as or similar to test creation subsystem 114, in accordance withone or more implementations.

In an operation 504, a second user input indicating a test performancebenchmark value for the test may be received. Operation 504 may beperformed by a test creation subsystem that is the same as or similar totest creation subsystem 114, in accordance with one or moreimplementations.

In an operation 506, a plurality of questions that correspond to thesubject matter, the question categories, and the intended questioneegroup may be obtained based on the first user input. Operation 506 maybe performed by a question management subsystem that is the same as orsimilar to question management subsystem 112, in accordance with one ormore implementations.

In an operation 508, performance information (comprising a questionperformance metric value associated with the question that is calculatedbased on prior performance of questionees with the question) may beobtained for each question of the plurality of questions. In animplementation, for each question of the plurality of questions, thequestion performance metric value may comprise at least one of (i) anamount of time estimated for a questionee to correctly answer thequestion or (ii) an amount of time estimated for a questionee toincorrectly answer the question. The amount of time for a questionee tocorrectly answer the question may be estimated based on one or moreprior amounts of time that the questionees took to correctly answer thequestion. The amount of time for a questionee to incorrectly answer thequestion may be estimated based on one or more prior amounts of timethat the questionees took to incorrectly answer the question. In anotherimplementation, for each of the plurality of questions, the questionperformance metric value may comprise a question difficulty levelassociated with the question that is estimated based on the priorperformance of the questionees with the question. Operation 508 may beperformed by a test creation subsystem that is the same as or similar totest creation subsystem 114, in accordance with one or moreimplementations.

In an operation 510, a test performance metric value associated with thetest may be calculated based on at least some of the questionperformance metric values associated with at least some questions of theplurality of questions. In an implementation, the test performancemetric value may comprise an amount of time for a questionee to completethe test. The calculation of the test performance metric value maycomprise estimating the amount of time for a questionee to complete thetest based on at least one of (i) the amounts of time estimated for aquestionee to correctly answer the at least some questions or (ii) theamounts of time estimated for a questionee to incorrectly answer the atleast some questions. In another implementation, the test performancemetric value may comprise a test difficulty level associated with thetest. The calculation of the test performance metric value may compriseestimating the test difficulty level based on the question difficultylevels associated with the at least some questions. Operation 510 may beperformed by a performance assessment subsystem that is the same as orsimilar to test creation subsystem 118, in accordance with one or moreimplementations.

In an operation 512, the at least some questions may be selected asrecommended questions (for inclusion in the test) based on the testperformance benchmark value and the test performance metric value.Operation 512 may be performed by a test creation subsystem that is thesame as or similar to test creation subsystem 114, in accordance withone or more implementations.

In an operation 514, the at least some questions and the testperformance metric value may be provided for presentation on a userinterface. Operation 514 may be performed by a test creation subsystemthat is the same as or similar to test creation subsystem 114, inaccordance with one or more implementations.

In an operation 516, the test may be generated such that the testincludes the at least some questions. Operation 516 may be performed bya test creation subsystem that is the same as or similar to testcreation subsystem 114, in accordance with one or more implementations.

FIG. 6 is an exemplary illustration of a flowchart 600 of a method ofproviding presentation of a test performance metric value associatedwith a test, in accordance with an aspect of the invention.

In an operation 602, a plurality of questions and performance metricvalues associated with the plurality of questions may be provided forpresentation on a user interface. Operation 602 may be performed by atest creation subsystem that is the same as or similar to test creationsubsystem 114, in accordance with one or more implementations.

In an operation 604, a first user input indicating selection of at leastsome questions of the plurality of questions (as questions for inclusionin a test) may be received. Operation 604 may be performed by a testcreation subsystem that is the same as or similar to test creationsubsystem 114, in accordance with one or more implementations.

In an operation 606, a test performance metric value associated with thetest may calculated (based on the selection of the at least somequestions) using the question performance metric values associated withthe at least some question. Operation 606 may be performed by aperformance assessment subsystem that is the same as or similar toperformance assessment subsystem 118, in accordance with one or moreimplementations.

In an operation 608, the at least some questions and the testperformance metric value may be provided for presentation on the userinterface. Operation 608 may be performed by a test creation subsystemthat is the same as or similar to test creation subsystem 114, inaccordance with one or more implementations.

In an operation 610, a second user input indicating selection of anadditional question of the plurality of questions (as a question forinclusion in the test) may be received subsequent to the first userinput. Operation 610 may be performed by a test creation subsystem thatis the same as or similar to test creation subsystem 114, in accordancewith one or more implementations.

In an operation 612, an updated test performance metric value associatedwith the test may be calculated (based on the selection of theadditional question) using the question performance metric valuesassociated with the at least some questions and an additional questionperformance metric value associated with the additional question.Operation 612 may be performed by a performance assessment subsystemthat is the same as or similar to performance assessment subsystem 118,in accordance with one or more implementations.

In an operation 614, the updated test performance metric value may beprovided for presentation on the user interface. Operation 614 may beperformed by a test creation subsystem that is the same as or similar totest creation subsystem 114, in accordance with one or moreimplementations.

FIG. 7 is an exemplary illustration of a flowchart 700 of a method ofproviding a comparison of performance of a questionee with performanceof one or more other questionees, in accordance with an aspect of theinvention.

In an operation 702, a test may be provided to a questionee. As anexample, the provided test may comprise one or more questions from aplurality of questions that correspond to a subject matter, questioncategories related to the subject matter, a questionee group for whichthe test is intended, or other criteria selected by a user. Operation702 may be performed by a test administration subsystem that is the sameas or similar to test administration subsystem 116, in accordance withone or more implementations.

In an operation 704, first performance information associated withquestions of the test and the questionee may be obtained. Operation 704may be performed by a performance assessment subsystem that is the sameas or similar to performance assessment subsystem 118, in accordancewith one or more implementations.

In an operation 706, second performance information associated with atleast one question of the test and one or more other questionees may beobtained. Operation 706 may be performed by a performance assessmentsubsystem that is the same as or similar to performance assessmentsubsystem 118, in accordance with one or more implementations.

In an operation 708, a comparison indicating relative performance of thequestionee with respect to the at least one question may be generatedbased on the first performance information and the second performanceinformation. Operation 708 may be performed by a performance assessmentsubsystem that is the same as or similar to performance assessmentsubsystem 118, in accordance with one or more implementations.

Although the present invention has been described in detail for thepurpose of illustration based on what is currently considered to be themost practical and preferred implementations, it is to be understoodthat such detail is solely for that purpose and that the invention isnot limited to the disclosed implementations, but, on the contrary, isintended to cover modifications and equivalent arrangements that arewithin the scope of the appended claims. For example, it is to beunderstood that the present invention contemplates that, to the extentpossible, one or more features of any implementation can be combinedwith one or more features of any other implementation.

What is claimed is:
 1. A system for facilitating creation of aneducational test based on prior performance with individual testquestions, the system comprising: one or more physical processorsprogrammed to execute computer program instructions which, whenexecuted, cause the one or more physical processors to: generate a userinterface configured to receive a set of parameters for the test, theset of parameters comprising a subject matter of the test, one or morequestion categories for the test, and a questionee group for which thetest is intended; receive, via the user interface, a first user inputfrom a user indicating the subject matter, the one or more questioncategories, and the questionee group; obtain, based on the first userinput, a set of questions that correspond to the subject matter, the oneor more question categories, and the intended questionee group; obtain,for questions of the set of questions, performance informationassociated with the question, wherein the performance informationcomprises a question performance metric value associated with thequestion that is calculated based on prior performance of one or morequestionees with the question; display, via a user interface, aselectable listing of the set of questions; receive, via the selectablelisting of the user interface, a first selection of a first questionfrom among the set of questions, the first question having a firstquestion performance metric value; add the first question to the testbased on the first selection; calculate a test performance metric valueassociated with the test based on at least the first questionperformance metric value; provide the test performance metric value forpresentation on the user interface; receive, via the user interface, asecond selection of a second question from among the set of questions,the second question having a second question performance metric value;add the second question to the test based on the second selection;update the test performance metric value associated with the test basedon the second question performance metric value; provide the updatedtest performance metric value via the user interface; receive a seconduser input from the user indicating a target level of difficulty for thetest; determine a current level of difficulty for the test based on atleast the first question and the second question; identify one or morequestions each associated with its own level of difficulty, recommendedfor inclusion in the test based on the current level of difficulty ofthe test, and the level of difficulty for each of the one or morequestions such that inclusion of the one or more questions will resultin the target level of difficulty for the test being achieved; anddisplay a recommendation to include the identified one or more questionsin the test via the user interface.
 2. The system of claim 1, whereinthe plurality of questions comprises one or more questions thatcorrespond to a plurality of questionee groups, and wherein each of theplurality of questionee groups correspond to a different educationallevel.
 3. The system of claim 1, wherein the one or more physicalprocessor are further caused to: receive another user input from theuser indicating a test performance benchmark value for the test; andselect, based on the test performance benchmark value and the testperformance metric value, the at least some questions as questionsrecommended for inclusion in the test.
 4. The system of claim 3, whereinthe test performance benchmark value relates to a duration of the test.5. The system of claim 1, wherein the one or more physical processorsare further caused to: provide the plurality of questions and thequestion performance metric values associated with the plurality ofquestions for presentation on the user interface; receive a third userinput from the user indicating selection of the at least some questionsas questions for inclusion in the test; receive, subsequent to the thirduser input, a fourth user input from the user indicating selection of anadditional question of the plurality of questions as a question forinclusion in the test; calculate an updated test performance metricvalue associated with the test based on the question performance metricvalues associated with the at least some questions and an additional oneof the question performance metric values associated with the additionalquestion; and provide the updated test performance metric value forpresentation on the user interface.
 6. The system of claim 1, wherein,for each question of the plurality of questions, the questionperformance metric value comprises at least one of an amount of timeestimated for a questionee to correctly answer the question or an amountof time estimated for a questionee to incorrectly answer the question,and wherein the amount of time for a questionee to correctly answer thequestion is estimated based on one or more prior amounts of time thatthe one or more questionees took to correctly answer the question, andthe amount of time for a questionee to incorrectly answer the questionis estimated based on one or more prior amounts of time that the one ormore questionees took to incorrectly answer the question.
 7. The systemof claim 6, wherein the test performance metric value comprises anamount of time for a questionee to complete the test, and whereincalculating the test performance metric value comprises estimating theamount of time for a questionee to complete the test based on at leastone of (i) the amounts of time estimated for a questionee to correctlyanswer the at least some questions or (ii) the amounts of time estimatedfor a questionee to incorrectly answer the at least some questions. 8.The system of claim 1, wherein, for each of the plurality of questions,the question performance metric value comprises a question difficultylevel associated with the question that is estimated based on the priorperformance of the one or more questionees with the question.
 9. Thesystem of claim 8, wherein the test performance metric value comprises atest difficulty level associated with the test, and wherein calculatingthe test performance metric value comprises estimating the testdifficulty level based on the question difficulty levels associated withthe at least some questions.
 10. The system of claim 1, wherein, foreach question of the plurality of questions, the performance metricvalue is calculated with respect to the intended questionee group, andwherein each of the one or more questionees is a questionee of theintended questionee group at the time the questionee of the intendedquestionee group answered the question.
 11. The system of claim 1,wherein the one or more physical processors are further caused to:provide the test to a first questionee; obtain first performanceinformation associated with the at least some questions and the firstquestionee, wherein the first performance information comprises, foreach question of the at least some questions, a first questionperformance metric value associated with the question and the firstquestionee; and update, for at least one question of the at least somequestions, the question performance metric value associated with the atleast one question based on the first performance metric valueassociated with the at least one question and the first questionee. 12.The system of claim 1, wherein the one or more physical processors arefurther caused to: provide the test to a first questionee; obtain firstperformance information associated with the at least some questions andthe first questionee, wherein the first performance informationcomprises, for each question of the at least some questions, a firstquestion performance metric value associated with the question and thefirst questionee; obtain second performance information associated withat least one question of the at least some questions and one or moreother questionees; and generate, based on the first performanceinformation and the second performance information, a comparisonindicating performance of the first questionee relative to performanceof one or more other questionees with respect to the at least onequestion.
 13. The system of claim 1, wherein the one or more physicalprocessors are further caused to: provide the test to a first questioneeassociated with a first entity, wherein the first entity is an entityother than the first questionee; obtain first performance informationassociated with the at least some questions and the first questionee,wherein the first performance information comprises, for each questionof the at least some questions, a first question performance metricvalue associated with the question and the first questionee; obtainsecond performance information associated with at least one question ofthe at least some questions and one or more other questionees, whereinthe one or more questionees are associated with one or more otherentities other than the one or more questionees; and generating, basedon the first performance information and the second performanceinformation, a comparison indicating performance of the first entityrelative to performance of the one or more other entities with respectto the at least one question.
 14. The system of claim 13, wherein thefirst entity is an instructor of the first questionee, and the one ormore other entities are one or more instructors of the one or more otherquestionees, and wherein the generated comparison indicates performanceof the instructor of the first questionee relative to performance of theone or more instructors of the one or more other questionees withrespect to the at least one question.
 15. The system of claim 13,wherein the first entity is an institution of the first questionee, andthe one or more other entities are one or more institutions of the oneor more other questionees, and wherein the generated comparisonindicates performance of the institution of the first questioneerelative to performance of the one or more institutions of the one ormore other questionees with respect to the at least one question.
 16. Amethod for facilitating creation of an educational test based on priorperformance with individual test questions, the method being implementedon a computer system that includes one or more physical processorsexecuting computer program instructions which, when executed, cause theone or more physical processors to perform the method, the methodcomprising: generating, by the computer system, a user interfaceconfigured to receive a set of parameters for the test, the set ofparameters comprising a subject matter of the test, one or more questioncategories for the test, and a questionee group for which the test isintended; receiving, by the computer system, via the user interface, afirst user input from a user indicating the subject matter, the one ormore question categories, and the questionee group; obtaining, by thecomputer system, based on the first user input, a set of questions thatcorrespond to the subject matter, the one or more question categories,and the intended questionee group; obtaining, by the computer system,for questions of the set of questions, performance informationassociated with the question, wherein the performance informationcomprises a question performance metric value associated with thequestion that is calculated based on prior performance of one or morequestionees with the question; displaying, by the computer system, via auser interface, a selectable listing of the set of questions; receiving,by the computer system, via the selectable listing of the userinterface, a first selection of a first question from among the set ofquestions, the first question having a first question performance metricvalue; adding, by the computer system, the first question to the testbased on the first selection; calculating, by the computer system, atest performance metric value associated with the test based on at leastthe first question performance metric value; and providing, by thecomputer system, the test performance metric value for presentation onthe user interface; receiving, by the computer system, via the userinterface, a second selection of a second question from among the set ofquestions, the second question having a second question performancemetric value; adding, by the computer system, the second question to thetest based on the second selection; updating, by the computer system,the test performance metric value associated with the test based on thesecond question performance metric value; providing, by the computersystem, the updated test performance metric value via the userinterface; receiving, by the computer system, a second user input fromthe user indicating a target level of difficulty for the test;determining, by the computer system, a current level of difficulty forthe test based on at least the first question and the second question;identifying, by the computer system, one or more question eachassociated with its own level of difficulty, recommended for inclusionin the test based on the current level of difficulty for the test, thetarget level of difficulty of the test, and the level of difficulty foreach of the one or more questions such that inclusion of the one or morequestions will result in the target level of difficulty being achieved;and displaying, by the computer system, a recommendation to include theidentified one or more questions in the test via the user interface. 17.The method of claim 16, wherein the plurality of questions comprises oneor more questions that correspond to a plurality of questionee groups,and wherein each of the plurality of questionee groups correspond to adifferent educational level.
 18. The method of claim 16, wherein themethod further comprises: receiving, by the computer system, anotheruser input from the user indicating a test performance benchmark valuefor test; and selecting, by the computer system, based on the testperformance benchmark value and the test performance metric value, theat least some questions as questions recommended for inclusion in thetest.
 19. The method of claim 18, wherein the test performance benchmarkvalue relates to at least one of a duration of the test or a testdifficulty associated with the test.
 20. The method of claim 16, whereinthe method further comprises: providing, by the computer system, theplurality of questions and the question performance metric valuesassociated with the plurality of questions for presentation on the userinterface; receiving, by the computer system, a third user input fromthe user indicating selection of the at least some questions asquestions for inclusion in the test; receiving, by the computer system,subsequent to the third user input, a fourth user input from the userindicating selection of an additional question of the plurality ofquestions as a question for inclusion in the test; calculate an updatedtest performance metric value associated with the test based on thequestion performance metric values associated with the at least somequestions and an additional one of the question performance metricvalues associated with the additional question; and provide the updatedtest performance metric value for presentation on the user interface.21. The method of claim 16, wherein, for each question of the pluralityof questions, the question performance metric value comprises at leastone of (i) an amount of time estimated for a questionee to correctlyanswer the question or (ii) an amount of time estimated for a questioneeto incorrectly answer the question, and wherein the amount of time for aquestionee to correctly answer the question is estimated based on one ormore prior amounts of time that the one or more questionees took tocorrectly answer the question, and the amount of time for a questioneeto incorrectly answer the question is estimated based on one or moreprior amounts of time that the one or more questionees took toincorrectly answer the question.
 22. The method of claim 16, wherein,for each of the plurality of questions, the question performance metricvalue comprises a question difficulty level associated with the questionthat is estimated based on the prior performance of the one or morequestionees with the question.
 23. The method of claim 16, wherein, forat least the first question, the first question performance metric valueis calculated with respect to a first intended questionee group based onperformance on the first question by first questionees in the firstintended questionee group that answered the first question, the methodfurther comprising: generating, by the computer system, another questionperformance metric value, different from the first question performancemetric value, for the first question with respect to a second intendedquestionee group based on performance on the first question by secondquestionees in the second intended questionee group that answered thefirst question such that performance of the first question is assesseddifferently for different questionee groups that answered the firstquestion.
 24. The system of claim 1, wherein the plurality of parametersfurther comprise a percentage of each of the one or more questioncategories that should make up the test, and wherein to obtain theplurality of questions, the one or more physical processors are furthercaused to: obtain a first percentage of a first question category ofquestions based on the percentage of each of the one or more questioncategories that should make up the test; and obtain a second percentageof a second question category of questions based on the percentage ofeach of the one or more question categories that should make up thetest.
 25. The system of claim 1, wherein the one or more physicalprocessors are further caused to: obtain a predefined composition ofcategories of subject matter to be provided to the intended questioneegroup; identify a set of questions to be added to the test based on thepredefined composition of categories of subject matter; and provide arecommendation to add one or more questions from the set of questions.26. The system of claim 25, wherein to obtain the predefined compositionof categories of subject matter, the one or more physical processors arefurther caused to: obtain a syllabus provided to students who aremembers of the intended questionee group.
 27. The system of claim 25,wherein to obtain the predefined composition of categories of subjectmatter, the one or more physical processors are further caused to:obtain homework assignments provided to students who are members of theintended questionee group, wherein categories of questions in the homeassignments are used as a basis for the predefined composition ofcategories of subject matter.
 28. The system of claim 25, wherein toobtain the predefined composition of categories of subject matter, theone or more physical processors are further caused to: obtain quizzesprovided to students who are members of the intended questionee group,wherein categories of questions in the quizzes are used as a basis forthe predefined composition of categories of subject matter.
 29. Thesystem of claim 1, wherein the user interface is further configured toreceive a new question, and wherein the one or more physical processorsare further caused to: add the new question to the test; add the newquestion to a question database from which the plurality of questionswere obtained; monitor answers to the new question; and generate a newquestion performance metric value for the new question based on themonitored answers.